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STATE FULL TEST METHOD EXECUTOR 

Background of the Invention 

5 Test automation systems are used to automatically test software-driven 

systems. Conventional architectures use a test harness, which is typically used to 
execute test cases, including test cases specified by attributes. The attributes are used to 
label classes or methods to mark which of the classes or methods should be examined 
for testing. 

1 0 Conventional test harnesses operate on test cases specified by attributes 

by examining each of the labeled classes or methods, examining the attributes present, 
and then performing an action in response to the examined attributes. This method of 
using attributes requires that the code that is used to perform the action to be present in 
the test harness. Accordingly, a test harness must be modified (or written) to extend or 

1 5 create different test case formats used to test differing software applications or various 
hardware/software combinations that may be present in a system. Additionally, each 
automation system will also need to have its associated test harness modified if 
supporting the test format used by another automation system is desired. 

Modifying or writing a test harness for different test case formats 

20 typically requires writing or modifying a non-trivial amount of code in each test harness 
so that the test harness can evaluate the test results. The implemented code for each test 
case format includes code to gather data about the test method being executed (e.g., test 
method description, test method attributes list, and the like), code to process the results 
from the attributes, and code that is used to determine the actual result of the test 

25 method. 

In accordance with the present invention, a test harness can be reused 
without requiring modifications to run tests written using different test case formats. 
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Summary of the Invention 

The present invention is directed towards a test method executor, which 
is an object designed to execute a test method. The test method executor typically 
implements a simple state engine that is used to process the data presented by each 
5 smart test attribute. The test method executor invokes particular test methods in 
response to the processed data. The test method executor then condenses the results 
from all the attributes typically into a simple return value, which is designed to be 
interpreted by the test harness. The return value indicates, for example, whether the 
executed test method was a success, failure, or if the test method was "skipped" 

10 Additionally, the test method executor is used to gather data about the method being 
invoked (such as the method name, which attributes are present, and the like.) 

According to one aspect of the invention, a computer-readable medium 
having computer- executable components comprises three components. A test case 
scenario object comprises a test method that is arranged to test an electronic system. A 

1 5 test harness is arranged to invoke the test method by issuing a request to invoke the test 
method. A test method executor is arranged to receive the request and invoke the 
requested test method by evaluating attributes that are associated with the requested test 
method before execution of the requested test method, to execute the requested test 
method, to evaluate attributes of the executed test method, and to return a result to the 

20 test harness that signals the result of the evaluation of the executed test method 
attributes. 

According to another aspect of the invention, a method for executing test 
components comprises receiving an invocation request to invoke a test method for 
testing an electronic system. Attributes that are associated with the requested test 

25 method are evaluated such that the attributes are evaluated before execution of the 

requested test method and that the attributes are evaluated in response to the invocation 
request. The requested test method is executed in response to the evaluation of the 
attributes. The attributes that are associated with the executed test method are 
evaluated. A result value is determined that signals the result of the evaluation of the 

30 executed test method attributes. 
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According to yet another aspect of the invention, a system for automated 
testing comprises three components. A test case scenario object comprises a test 
method that is arranged to test an electronic system. A test harness is arranged to 
invoke the test method by issuing a request to invoke the test method. A test method 
5 executor is arranged to receive the request and invoke the requested test method by 
evaluating attributes that are associated with the requested test method before execution 
of the requested test method, to execute the requested test method, to evaluate attributes 
of the executed test method, and to return a result to the test harness that signals the 
result of the evaluation of the executed test method attributes. 

10 According to a further aspect of the invention, a system for automated testing 

comprises five means. Means for receiving an invocation request to invoke a test 
method for testing an electronic system. Means for evaluating attributes that are 
associated with the requested test method such that the attributes are evaluated before 
execution of the requested test method and that the attributes are evaluated in response 

1 5 to the invocation request. Means for executing the requested test method in response to 
the evaluating of the attributes. Means for evaluating attributes that are associated with 
the executed test method. Means for determining a result value that signals the result of 
the evaluation of the executed test method attributes. 

20 Brief Description of the Drawings 

FIGURE 1 illustrates an exemplary computing device that may be used 
in one exemplary embodiment of the present invention. 

FIGURE 2 is a block diagram illustrating an exemplary environment for 
practicing the present invention. 
25 FIGURE 3 illustrates of a process 300 flow of an execution engine, in 

accordance with aspects of the invention. 

FIGURE 4 is a further illustration of a process 400 flow of an execution 
engine, in accordance with aspects of the invention. 
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Detailed Description of the Preferred Embodiment 

Throughout the specification and claims, the following terms take the 
meanings explicitly associated herein, unless the context clearly dictates otherwise. The 
terminology and interface specifications used herein are not intended to represent a 
5 particular language in which a particular object or method should be written. Rather, 
the terminology and interface specifications are used to describe the functionality and 
contents of an interface or object, such as function names, inputs, outputs, return values, 
and what operations are to be performed using the interface (or what operations are to 
be performed by the object). 

10 

Illustrative Operating Environment 

With reference to FIGURE 1, one exemplary system for implementing 
the invention includes a computing device, such as computing device 1 00. In a very 
basic configuration, computing device 100 typically includes at least one processing 

15 unit 102 and system memory 104. Depending on the exact configuration and type of 
computing device, system memory 104 may be volatile (such as RAM), non- volatile 
(such as ROM, flash memory, etc.) or some combination of the two. System memory 
104 typically includes an operating system 105, one or more applications 106, and may 
include program data 107. In one embodiment, application 106 may include a word- 

20 processor application 120 that further includes ML editor 122. This basic configuration 
is illustrated in FIGURE 1 by those components within dashed line 108. 

Computing device 100 may have additional features or functionality. 
For example, computing device 100 may also include additional data storage devices 
(removable and/or non-removable) such as, for example, magnetic disks, optical disks, 

25 or tape. Such additional storage is illustrated in FIGURE 1 by removable storage 109 
and non-removable storage 1 10. Computer storage media may include volatile and 
nonvolatile, removable and non-removable media implemented in any method or 
technology for storage of information, such as computer readable instructions, data 
structures, program modules, or other data. System memory 104, removable 

30 storage 109 and non-removable storage 1 10 are all examples of computer storage 
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media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, 
flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or 
other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other 
magnetic storage devices, or any other medium which can be used to store the desired 
5 information and which can be accessed by computing device 100. Any such computer 
storage media may be part of device 100. Computing device 100 may also have input 
device(s) 112 such as keyboard, mouse, pen, voice input device, touch input device, etc. 
Output device(s) 1 14 such as a display, speakers, printer, etc. may also be included. 
These devices are well know in the art and need not be discussed at length here. 

10 Computing device 100 may also contain communication connections 116 

that allow the device to communicate with other computing devices 118, such as over a 
network. Communication connection 1 16 is one example of communication media. 
Communication media may typically be embodied by computer readable instructions, 
data structures, program modules, or other data in a modulated data signal, such as a 

1 5 carrier wave or other transport mechanism, and includes any information delivery 
media. The term "modulated data signal" means a signal that has one or more of its 
characteristics set or changed in such a manner as to encode information in the signal. 
By way of example, and not limitation, communication media includes wired media 
such as a wired network or direct-wired connection, and wireless media such as 

20 acoustic, RF, infrared and other wireless media. The term computer readable media as 
used herein includes both storage media and communication media. 

The present invention is directed towards a test method executor, which 
is an object designed to execute a test method. The test method executor typically 
implements a simple state engine that is used to process the data presented by each 

25 smart test attribute. The test method executor invokes particular test methods in 

response to the processed data. The test method executor then condenses the results 
from all the attributes typically into a simple return value, which is designed to be 
interpreted by the test harness. The return value indicates, for example, whether the 
executed test method was a success, failure, or if the test method was "skipped." 



5 



Additionally, the test method executor is used to gather data about the method being 
invoked (such as the method name, which attributes are present, and the like.) 

In accordance with the present invention, a test automation system is 
presented that comprises a test harness, a test method executor, and a test case scenario. 
5 The presented architecture allows the test harness to invoke a test method by simply 
calling a single "invoke" method on the test method executor. Additionally, the test 
harness is able to gather information (for display, logging, or other reasons) about the 
test methods called by the test method executor. The test harness can gather the 
information by calling a single method to retrieve information of the desired type. (This 

1 0 can be contrasted with conventional systems, wherein the conventional test harness was 
required to implement a particular method within the test harness for each type of 
information desired.) 

Using a test method executor in accordance with the present invention 
allows for more consistent execution of a test (as compared to conventional test 

15 harnesses, in which a particular case may give differing results when run on differing 
automation systems). The test method executor also allows for a "thinner" test harness, 
which reduces code duplication that is present with conventional test harnesses, and the 
increases the (re)use of the code that is written for the "thinner" test harness of the 
present invention. The interaction between the conventional test harnesses and the test 

20 case scenario itself is reduced by the present invention to an abstraction layer that 

couples the method executor to the user interface. Furthermore, various test cases that 
are written in accordance with the present invention can be shared amongst various test 
harnesses (and their respective automation systems) without requiring rewriting or 
otherwise modifying the test cases. 

25 

Test Automation System Architecture 

FIGURE 2 is a block diagram illustrating an exemplary environment for 
practicing the present invention. The exemplary environment shown in FIGURE 2 is a 
test automation system 200 that comprises test harness 210, test runtime 220, and test 
30 case scenario 230. 
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Test Runtime 

In one embodiment, test runtime 220 is a collection of objects that 
abstracts knowledge of test cases from the test harness. Test runtime 220 typically 
5 comprises test services provider object 221, extraction engine 222, attributes 223, and 
test method executor 224. Test runtime 220 can be used by different harnesses to 
provide consistent support for a particular type of test case format. By extension, a test 
harness can use different test runtimes to support different types of test case formats. 
The test harness typically determines which test runtime to use for a particular test case 
10 format. 

Test case extraction is accomplished through a dedicated extraction 
object (obtained from a test services provider object 221, described below), and 
invocation is performed by an object (test method executor 224, also described below), 
which returns a generic result to the test harness. The test method executor evaluates 

1 5 attributes using a generic interface to control the execution of the method. 

A test harness is not required to, for example, evaluate attributes, 
determine the order that in which test methods should executed, build argument lists for 
method invocations, and the like. The test harness typically does not have direct access 
to this information required to perform those tasks, which helps ensure more consistent 

20 test execution between different test harnesses. All test harness dependent functionality 
(logging, "remoting," and the like) should be objects implemented by the test harness, 
described by an interface, and stored in the test service provider object for use during 
test execution. Accordingly, a test harness can be created which is easily capable of 
switching between different test runtimes without requiring changes to the test harness 

25 code. 



Test Services Provider 

Test services provider object 221 is used by the test harness to retrieve 
an extraction engine and is also used by the test case scenario to retrieve objects for test 
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harness-implemented functionality (including test services such as logging, 
synchronization, and the like). 

The test services provider object typically provides methods that are used 
to facilitate access to test services. An AddService method is called to store references 
5 to objects that implement a test service. A test service object should implement a 
"generic" type, which facilitates a set of standard services that different harnesses can 
provide. The object passed in should implement functionality needed by attributes or 
test methods. This method should be called by the test harness for each service the test 
harness provides, which is generally done before the test methods are executed. 

10 A GetService method is typically called to retrieve a type of test service 

object. The type passed in should represent an interface implemented by a test service 
object. An object that implements that object will be returned if found. If the object is 
not found, a value is returned that indicates such a result (e.g., returns a null). If a test 
or attribute requires a test service that is not present, the test method being executed 

15 should fail. 

An AddDefaultServices protected method is typically called by the 
constructor. It is typically used to add whatever test services the test runtime provides 
by default, such as an extraction engine. 

The test services provider object data and method calls should be static 
20 such that the data set by the test harness can be retrievable by other calls to the test 

service provider at a point later in time. The object is typically a class that implements 
the functionality described above. 

Extraction Engine 

25 Extraction engine 222 is used to retrieve an ordered list of test methods 

from the test case for the test harness for a particular test case scenario. Typically there 
is only one extraction engine present in a test runtime. 

A GetTestMethodWrappers method is used to retrieve the ordered list of 
test methods for a test case scenario. (A test method wrapper is a particular 

30 implementation of a test method executor.) A parameter is passed into the method that 
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represents a container that holds test cases. The method returns the ordered list of 
methods. If an error occurs while extracting the test case, an exception can be 
"thrown." If no test cases or test methods are found, an empty list is normally returned. 
An empty list is usually treated as a failure by the test harness. If additional data needs 
5 to be passed to the extraction engine, it can be provided through the 
TestServicesProvider object by a TestHarnessDataProvider test service. 

A large part of the functionality present in an extraction engine object is 
the same between different runtimes; the only detail that may change significantly from 
one runtime to another is which attributes are used. An object should be created in 
10 order to facilitate easy creation of new extraction engines and runtimes. This class 
typically implements the following functions: 

TypelsATestCase returns a "true" if the type passed in is a test case, 
"false" if not. This function examines the attributes present on a type by looking for a 
test case attribute to determine if a type is a test case or not. 
15 MethodlsATestMethod returns true if the method passed in is a test 

method, false if not. This function examines the attributes present on a type by looking 
for an execution attribute to determine if a type is a test method or not. 

GetMethodWrappersFromType gathers all relevant public non-static 
methods on a type that have execution attributes and returns the methods as an ordered 
20 list of Method Wrappers (discussed below). In an embodiment using the .Net 

environment, the list is ordered using the sort functionality built into .Net arrays so that 
the Method Wrapper calls a compare routine to order the list. If an instance of the type 
cannot be created, this method fails and returns an empty list. 

The extraction engine can use a test service to retrieve information that 
25 can be used to modify a test extraction. The information can be stored in a file such as 
an XML file that is in accordance with a schema defined for storing test extraction 
information. If the data provided by the test service does not refer to an XML file in 
accordance with the schema, the modification data can be ignored. 

The extraction engine typically loads all extraction modifier XML files 
30 specified. The contents of the XML files are, for example, placed into two "buckets:" 
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test inclusions, and test exclusions. If both buckets are empty, the extraction engine 
should include all tests. This case is equivalent to having no extraction modifier xml 
files, or not being able to retrieve the Extraction Engine Data Source test service. If 
only the exclusion bucket is empty, the extraction engine should include all tests. If 
5 only the inclusion bucket is empty, the extraction engine should include all tests, and 
exclude tests listed in the exclusion bucket. If both buckets have data, the extraction 
engine should include tests in the inclusion bucket that are not listed in the exclusion 
bucket (such that the exclusion list has controlling authority over the inclusion list). 

10 Test Method Executors 

Test method executors 224 are used to execute a test method without 
requiring the caller to have beforehand knowledge about the method or its attributes. 
An Invoke method is called to execute a test method. An object holding the result of 
the operation is returned (pass, fail, skip, and the like). The Invoke method is 
1 5 responsible for processing the attributes associated with a method and creating a 
parameter list (if required) for the method being invoked. Execution is typically 
modified by the attributes associated with the test method. 

An Abort method can be called to abort a currently executing test 
method. The abort typically causes a currently running Invoke method to return. After 
20 an abort is performed, no further tests can normally be run. 

A CompareTo method is called to compare two Test Method Wrappers. 
If the result returned is less than zero, it indicates that this method should be executed 
before the other Test Method Wrapper (to which the method is compared). If the result 
returned equals zero it indicates that the order in which both methods are executed in 
25 does not matter. If the result returned is greater than zero, it indicates that this method 
should be executed after the other Test Method Wrapper. 

A GetMethodAttributes method is called to retrieve sorted list of 
attributes associated with the test method, which are derived from a common base 
method attribute class. This sorted list of attributes is used by the Test Method Wrapper 
30 in several locations. For example, the Invoke method (as discussed above) uses the 
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sorted list to evaluate the attributes in the correct order. Also, the list returned by 
GetMethodAttributes can be used to compare one method wrapper to another. A call 
such as "get AttributesDescription" uses the ordered list of attributes to create a string 
description of the attributes associated with the method. 
5 The test method wrapper has several properties that can be retrieved for 

use by the test harness, such as by calling "get Description" and "get Name." The 
properties are generated from the name of the method and the class from which the 
method was defined. Other properties can be added as needed. Note that these 
properties do not require the harness to know anything about the test method being 

10 queried and that additional properties can be added without requiring modifications to 
existing test harnesses. 

The MethodResult object is used to convey the result of a test method 
wrapper to the test harness. Because the test harness does not necessarily have 
beforehand knowledge of the method being invoked, the results are expressed in 

15 abstract form. 

The object typically needs to express three possible outcomes from 
trying to execute a method: pass, skip, or fail. A "pass" would typically indicate that 
the method completed execution without any errors (for example, the method logged no 
failures, the test method did not throw an exception, and none of the attributes had an 

20 error). An "error" would indicate that the method failed (for example, the test method 
indicated a failure, or an attribute indicated a failure). A "skip" would indicate that the 
method was skipped rather than executed (for example, an attribute specifies the test 
method should only run on a server, but the test is running on a client machine; in which 
case the method would be skipped). 

25 The MethodResult object can also contain optional messages such as a 

result message and/or an error message. The result message can be a human readable 
description of the methods result. Upon a successful execution of a method, this could 
be left blank, or it could contain the number of passes recorded. For a completed 
method in which an error occurred, a textual description of the error may be included 

30 while the error message can contain the details of the error. 

11 



When an exception is "thrown" by an attribute a TestConditionException 
class can be used to convey a modified method state. For example, three derived 
classes that map directly to a method state include TestSkipException, 
TestSucceededException, and TestErrorException. 

Attributes 223 are typically used to modify and control the execution of a 
test. A test is executed according to a test case scenario, which can be defined through 
the use of attributes. At least three basic types of attributes can be used: class level 
attributes, method level attributes, and parameter level attributes. 

Test class attributes are optional and can be used to modify an 
instantiated object state, such that test extraction can be skipped or caused to be 
performed multiple times for a denoted type. Pre- and post-extraction methods are 
typically used to modify the instantiated object state. Test class attributes allow such 
variations in test case scenarios to be implemented. 

Method level attributes are capable of modifying method parameters and 
method execution. In an embodiment, method level attributes include execution 
attributes and supplemental attributes. Both attributes have pre- and post-invocation 
methods. The order in which method level attributes are evaluated is determined by an 
order property, which is defined when the attribute is written; however, attributes 
typically have no beforehand knowledge of what other attributes may be present. 
Execution modification at each stage can be handled by a priority based state system - 
the attribute returning a state with the highest priority is typically used to determine how 
execution is modified. 

Execution attributes are used to mark a method as a test method. A 
method without an execution attribute is usually not included in a test. The core 
responsibility of an execution attribute is to establish a high-level order to the test and to 
evaluate method results. A method should not have more than one execution attribute. 
Extraction and execution behavior when more than one execution attribute is present is 
normally undefined. Examples of execution attributes include "Setup", "Step", and 
"Teardown" attributes. 
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Supplemental attributes perform supplemental actions to modify the 
execution of a test method. The core responsibility of a supplemental attribute is to 
perform secondary tasks that are necessary for the execution of a test. They typically 
are not used to denote high-level order. A method may have any number of 
supplemental attributes. Examples of supplemental attributes include "WaitFor" and 
"Target" attributes. 

Parameter level attributes are optionally used to modify the parameter 
input to a method and to modify a state of an object (e.g., the context state) after a 
method has executed. A parameter level attribute is normally not used to alter the 
execution of a test as method level attributes are used. However, if an exception is 
thrown, the test method fails immediately in response to the exception. In an 
embodiment, there is only one parameter level attribute per parameter; behavior with 
more than one parameter level attribute is left undefined. Parameter level attributes are 
evaluated after method level attributes are evaluated before a method is invoked, and 
before method level attributes are evaluated after a method is invoked. An example of a 
parameter level attribute includes the "ContextMapping" attribute. 

The Methods tate object is used by method attributes to control the 
execution of a test method. Because a plurality of attributes can be assigned to a test 
method (and because each attribute can potentially alter the execution of the test 
method), each attribute can communicate with the MethodState object to ensure 
consistent execution of the test method. 

The MethodState object can comprise information related to the 
execution state, a message, an error code, and a state override priority. The execution 
state comprises information regarding how the method has terminated (e.g., skip, pass, 
fail), whether the state is permitted to be changed, and whether the method should be 
executed. The message can be used to optionally present text that indicates why the test 
method is in a particular state. The error code can be used to indicate the details of an 
error that the test method wrapper might encounter while executing a test method. The 
state override priority field can be used to improve the consistency of test method 
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execution by allowing the execution state, message, and the error code to be altered 
only if the new state has a priority greater than the existing state. 

The test method wrapper (224) executes a test method until a terminating 
state is reached. When the terminating state is reached, the MethodResult object is 
constructed from the final MethodState object. 

Execution attributes are responsible for parsing the result obtained from 
a method invocation. To determine if a method passed or failed, logs can be monitored 
for pass and fail entries. If any failures are logged, the method likely failed. If no 
passes or failures were logged, the method also likely failed. If an exception was 
thrown from the test method or any attributes, the method again likely failed. 
Otherwise the method can be considered to have (successfully) passed. 

Test Case Scenario 

In one embodiment, test case scenario 230 is a collection of objects that 
coordinate the execution of test methods for a particular test case. The test methods can 
be written without beforehand knowledge of the test harness because of the interface 
provided by and through the test method executor (224) and the collection of objects 
provided by/through the test runtime. 

Test case scenario 230 typically comprises test methods 231 and other 
methods and data 232. Test methods access test harness objects by using the test 
services provider object (which is comprised by the test runtime), rather than by 
querying specific test harnesses. 

Test Harness 

In one embodiment, test harness 210 is a collection of objects that 
coordinate the execution of test cases and provides various test services. Test harness 
210 typically comprises a UI (User Interface) 21 1, an Execution Engine 212, a Context 
object 213, and a Logging object 214. The test harness for purposes of added 
functionality may comprise other objects such as an automation system interface. 
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The execution engine (212) is responsible for loading and executing test 
case scenarios using the test runtime (220). FIGURE 3 illustrates a process 300 flow of 
an execution engine, in accordance with aspects of the invention. After a start block, 
the process moves to block 3 10, at which point a test runtime is loaded. In an 
embodiment wherein the test runtime is written in .Net, the test runtime assembly and 
test case scenario assembly is loaded into an AppDomain. The test harness can display 
information such as the version of the .Net runtime loaded, or the version of the test 
runtime being used. 

At block 320, one or more test cases are loaded/compiled into memory. 
The test cases can be precompiled and loaded into memory or loaded into memory and 
then compiled. The test harness can display information about the test case scenario as 
well as whether the test case scenario loaded and/or compiled successfully. 

Continuing at block 330, the extraction engine is obtained from the test 
runtime. The extraction engine is obtained by first retrieving the test services provider 
object (221). The static GetService function on the test services provider object is 
called (passing the type of the base extraction engine to the test services provider 
object) to receive a reference to an extraction engine. 

At block 340, test harness functionality is added to the test services 
provider. The AddService method on the test services provider is used to add to the test 
services provider the test services that are implemented by the test harness. The test 
services include objects that implement various interfaces such as the reboot 
mechanism, logging, context, and the like. If it is desirable to pass data to the extraction 
engine (such as, for example, an XML file to specify that certain method should be 
included or skipped), a test service object that implements a test harness data provider 
interface can be used. 

At block 350, the extraction engine is used to get test case steps. The 
extraction engine obtained at block 330 is used to call a GetTestMethodWrappers 
method of the extraction engine, passing to the method the AppDomain holding the test 
case scenario. An array of TestMethod Wrappers is typically returned. 
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The array of method wrappers typically contains a list of test actions that 
should be executed in the order in which they are present in the array. (The extraction 
engine is typically responsible for ordering the array). 

At block 360, the list of methods retrieved at block 350 is executed in the 
5 order in which the methods are listed. Each method is executed by calling the Invoke 
method. The invoke method typically returns details about the result of the test action. 
The details may include a success/fail/skip result, as well as additional details. The 
details of the result may be, for example, logged, or used to update the UL 

FIGURE 4 is a further illustration of a process 400 flow of an execution 
10 engine, in accordance with aspects of the invention. For each test action, process 400 is 
repeated. The execution engine calls the InvokelnstanceMethod to initiate the 
execution of a particular test action. The InvokelnstanceMethod calls the 
InstanceMethod to invoke the particular test action. The InstanceMethod in turn calls 
the Method Wrapper (i.e., an example test method executor) to invoke the particular test 
15 action. 

The method wrapper evaluates and executes the attributes (pre- 
invocation) of the particular test action. The method wrapper next invokes the test 
method in accordance with the evaluated/executed test method. After the test method 
has been executed, the attributes are again evaluated/executed (post-invocation). The 
20 method wrapper construes the result of the post-invocation attribute evaluation and 
returns a value that that signals the result. The ActionResult is passed to the 
InstanceMethod, and to the InvokelnstanceMethod in turn. The InvokelnstanceMethod 
evaluates the return value and passes the result to the execution engine. 

25 

As a further illustration of the invention, a listing of pseudocode is 
provided as follows: 

ExtractionEnaine-GetTestMethodWrappers 

30 TestMethodWrappers [] methodWrappers ; 
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For ( each assembly in AppDomain ) 

{ 

For ( each type in assembly ) 
5 { 

If ( Type I sATest Case (type) ) 

{ 

methodWrappers += 
GetMethodWrappersFromType (type) 
10 } 

} 

} 

Return methodWrappers 
15 ExtractionEnqine. GetMethodWrappersFromTy pe 

TestMethodWrappers [] allMethodWrappers 



ExtractionState cur St ate = invalid state 
Do 



20 { 



TestMethodWrappers [ ] instanceMethodWrappers 
curState .Reset ( ) 



if ( type is creatable ) 
25 Object testObject = new type; 

Else 

Return empty list; 

For ( each attribute on type ) 
30 { 

If ( attribute is kind of test case attribute ) 
curState = Attribute . PreExtract (testObject ) 

35 If ( curState = terminating state ) 

break; 

Methodlnfo [] potent ialTestMethods = 
type .GetAllPublicMethods () 
40 For ( each method in potentialTestMethods ) 

{ 

If ( Me thodlsATestMethod (method) ) 
{ 
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TestMethodWrapper wrapper = new 
Wrapper (obj , method) 

If ( 

! wrapper . GetMethodAttributes [0] .Inherit ) 
5 { 

If ( method. definingType == type ) 

instanceMethodWrappers += wrapper 

} 

Else 

10 instanceMethodWrappers + = wrapper 

} 

For ( each attribute on type ) 
15 { 

If ( attribute is kind of test case attribute ) 
curState = Attribute . PostExtract ( ) 

} 

20 instanceMethodWrappers . Sort ( ) 

allMethodWrappers += instanceMethodWrappers 

} 

While ( curState != terminating state ) // invalid state 
or skip 

25 

Return allMethodWrappers; 
TestMethod Wrapper-Invoke 

MethodState curState = invalid state 
Do 
30 { 

curState . Reset ( ) 

TestMethodAt tribute [] methodAttributes = 
GetMethodAttributes () 
35 methodAttributes . Reverse ( ) 

// pre -invoke for all attributes 
Try 

{ 

40 // method level attributes 

For ( each attribute in methodAttributes ) 

{ 

MethodState newState = 
attribute . Prelnvoke (...) 

18 



if ( newState . execstate != nochange && 

newState. priority > curState .priority 

) 
{ 

5 curState = newState 

} 

} 

} 

Catch (UnexpectedCondition) 
10 { 

curState = new MethodState (UnexpectedCondition) ; 

} 

Catch (e) 

{ 

15 // if an attribute throws an exception, the 

invoke fails 

curState = new MethodState (e) ; 
break; 

} 

20 

// evaluate state and stop invoke if necessary 
state must 

// be evaluated before examining parameter level 
attributes 

25 if ( curState. execstate == terminating state ) 

{ 

break; 

} 

30 Try 
{ 

// parameter level attributes these 
attributes can't 

// modify state, but if they throw the invoke 

35 fails 

For ( each parameter on the method ) 

{ 

For ( each attribute on the parameter ) 

{ 

40 If ( attribute is kind of 

Motif ParamAt tribute ) 

Attribute . Prelnvoke (...) 



} 

45 } 

Catch (e) 



} 
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{ 

curState = new MethodState (e) 
break; 

} 

5 

// execute test method 

Object result; 

Try 

{ 

10 result = method, invoke (...) 

} 

Catch (e) 

{ 

// note that if the method throws, the test does 
15 // not automatically fail; instead the exception 

is 

// treated as a return value for the execution 

attribute 

// to deal with 
20 result = e 

} 

// post -invoke on all attributes 
// if any attribute throws, the invoke fails 
25 curState. Reset () 

Try 

{ 

// post invoke for parameter level attributes 
For ( each parameter on the method ) 
30 { 

For each ( attribute on the parameter ) 

{ 

If ( attribute is kind of 
Motif ParamAt tribute ) 
35 Attribute . Postlnvoke (...) 

} 

} 

// post -invoke on all method level attributes 
40 For ( each attribute in methodAttributes ) 

{ 

MethodState newState = at trib. Postlnvoke (...) 
If ( newState . execstate != nochange && 

newState. priority > curState . priority 



45 ) 



{ 

20 



curState = newState; 

} 

} 

} 

5 Catch (UnexpectedCondition) 

{ 

curState = new MethodState (UnexpectedCondition) ; 

} 

Catch (e) 
10 { 

curState = new MethodState (e) 
break; 

} 

} 

15 While ( curState != terminating state ) // success, 

error, 

// skip, invalid state 

Log final state 

Return MethodResult (curState) ; 

20 ExecutionAttribute.Prelnvoke 

logManager = TestServiceProvider . GetService (LogManager) 
If ( logManager == null ) 

Throw UnexpectedErrorCondit ion ( "LogManager object not 
found") 

25 logManager . AddListener (us) 

Return MethodState (execute, low priority) 
ExecutionAttribute.Postlnvoke 
LogManage r . RemoveL i s t ene r ( us ) 
If ( result is kind of exception ) 

Return MethodState (failed, low priority) 
35 Else If ( failedCount > 0 ) 

Return MethodState ( failed, medium priority) 
Else If ( passedCount == 0 ) 

Return MethodState (failed, medium priority) 
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Else // passedCount > 0, failedCount = 0 

{ 

Return MethodState (pass , medium priority) 

} 

5 TargetAttribute.Prelnvoke 

If ( Context . CurrentMachine not in RunOnMachinesList ) 

Throw UnexpectedSkipCondition (Context . CurrentMachine 
+ w not in " + 

RunOnMachinesList) 

10 Else 

Return MethodState (nochange) 
VariationsAttribute.Postlnvoke 

currentVariation++ ; 

If ( currentVariation < totalVariations ) 
15 Return MethodState (execute, high priority) 

Else 

Return MethodState (pass/fail , medium priority) 

The above specification, examples and data provide a complete description of 
20 the manufacture and use of the composition of the invention. Since many embodiments 
of the invention can be made without departing from the spirit and scope of the 
invention, the invention resides in the claims hereinafter appended. 
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